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Abstract  -  Underwater  imaging  of  the  air  environment  is  described  with  particular  attention  given  to  the  optical  problems  of  near¬ 
horizon  viewing.  These  include  severe  chromatic  aberration  from  transmission  through  the  seawater  wedge,  the  horizon  distortion  effect, 
and  anisoplanatism  resulting  from  the  change  of  refractive  aberrations  in  different  areas  of  view.  Various  optical  designs  or  schemes  are 
presented  that  correct  or  relieve  the  first  two  optical  problems.  Strategies  to  minimize  or  attempt  to  view  through  the  varying  refraction  of  the 
surface  are  also  suggested,  and  one  is  investigated. 

I.  Introduction 

The  appearance  of  the  aerial  half- sphere  from  underwater  has  long  been  a  matter  of  recreational  curiosity  and  educational 
interest.  It  also  figures  in  the  survival  for  fish  that  rely  on  jumping  or  squirting  water  at  in-air  prey  and  the  lure-casting  accuracy 
of  human  anglers  [1][2].  This  discussion  is  chiefly  about  imaging  near  the  horizon  or  at  angles  very  near  the  edge  of  the  Snell 
window  (the  Snell  cone).  Imaging  under  these  circumstances  is  difficult  to  impossible  compared  with  imaging  near  the  vertical, 
although  the  imaging  near  vertical  through  disturbed  water  is  itself  deemed  very  difficult.  Theoretical  calculations  have 
concentrated  on  the  calm-water  case  [3],  but  the  likelihood  of  encountering  completely  calm  conditions — even  in  sheltered  bays 
and  rivers — is  quite  limited. 

Imaging  near  the  air-horizon  is  necessary  because  so  much  of  what  happens  that  is  interesting  to  sea  observers,  either  immersed 
or  in  air,  is  near  the  horizon.  For  example,  a  huge  oil  tanker  or  cruise  ship  with  upper  works  extending  40  m  above  the  water 
would  occupy  angles  less  than  5  degrees  at  a  distance  of  only  about  460  m  (little  more  than  its  own  length).  Lesser  objects  that 
one  might  want  to  view  from  a  submerged  position  would  likely  occupy  only  the  first  few  degrees  from  the  horizon. 

II.  The  Optical  Problems 

A.  Wedge  Chromatic  Aberration 

The  Snell  cone,  because  it  is  tilted  at  an  angle  of  approximately  48  degrees  from  the  vertical,  induces  a  prism-like  spectrum  in 
the  incoming  ray  bundle.  It  is  as  if  this  bundle  has  traversed  a  prism  with  apical  angle  of  48  degrees.  At  510  nm  under  laboratory 
pressures  and  temperatures,  light  has  a  modeled  index  of  refraction  in  seawater  of  n510  =  1.342404  and  at  610  nm  its  index  is  n610 
=  1.338877.  The  spectrum  of  this  virtual  wedge  of  seawater  has  an  angular  spread  of 

0  =  n560  siiUf— l-sirUf— 1  =  0.226  degrees,  (1) 

V^6ioy  V^5ioy_ 

where  n560  is  the  index  at  560  nm,  the  center  of  the  considered  spectrum.  This  subset  of  the  spectrum  was  chosen  following  [4], 
since  a  visual  or  a  visual-filtered  spectrum  was  the  region  of  greatest  interest.  Fig.  1  shows  the  effect  of  the  seawater  wedge  at  the 
focus  of  a  notional  57.3-mm  lens  modeled  in  ZEMAX®  [5].  The  boxes  are  plotted  in  units  of  pm,  and  lens  focal  length  is  chosen 
so  that  1  mm  of  movement  in  the  focal  plane  (1000  pm)  is  a  convenient  1 -degree  shift  in  the  focal  plane.  Note  that  the  spectrum  is 
profound  at  small  angle,  at  a  horizon  angle  of  2  degrees  the  spectrum  length  is  3.5degrees.  The  incidence  angles  are  annotated  at 
left  and  it  is  obvious  that  the  length  of  the  spectrum  varies  little  from  an  incidence  angle  of  70  degrees  (20  degree  horizon  angle) 
to  88  degrees  (2  degrees  horizon  angle).  Thus,  if  a  lens  or  corrective  mechanism  can  be  found  that  cures  this  problem  at  one  angle, 
it  will  also  be  useful  over  a  large  field-of-view. 
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Figure  1.  Prism  color  from  transmission  through  virtual  prism  of  seawater  of  light  from  near  the  air  horizon. 

The  image  is  upside  down. 


Much  higher  dispersions  than  the  crown-like[6]  dispersion  of  seawater  exist  in  flint  glasses,  so  a  corrective  dispersion  is 
attainable  without  coming  close  to  total  internal  reflection.  In  other  words,  a  corrective  window  consisting  of  an  oppositely- 
oriented  prism  can  easily  be  designed  and  will  work  under  a  wide  variety  of  surface  tilts.  Such  a  window  is  designed  in  Fig.  2.  It 
uses  a  high-dispersion  dense-flint  glass,  Schott  SF1 1  [7],  for  the  negative  prism,  but  could  use  any  one  of  a  number  of  high- 
dispersion  glasses,  including  the  modern  eco-friendly  glasses  that  have  superseded  many  high-lead  materials  such  as  SF1 1. 


88  degrees  from  zenith 


85  degrees  80  degrees  70  degrees 


The  virtual  seawater-prism  angle  has  shifted  from  48  to  56  degrees  because  the  beam  is  slightly  deflected  by  the  prism 
combination  and  the  horizon  must  be  placed  in  the  field-of-view.  The  calculated  prism  angle  is  13.4  degrees  to  correct  the  water 
wedge  precisely.  To  increase  the  number  of  mounting  options,  however,  this  prism  was  specified  at  14  degrees.  The  lens  is  a 
simple  achromatic  Landscape  [8]  design  of  20-mm  focal  length,  hand-built  for  this  camera  because  commercially-available 
pinhole  lenses  had  insufficient  focal-plane  coverage.  The  ZEMAX  spot  diagram  of  this  layout  is  depicted  in  Fig.  4.  It  is  corrected 
for  the  horizon  angle,  but  operates  acceptably  well  at  a  zenithal  angle  of  70  degrees. 

Figure  5  is  a  test  image  of  this  situation.  The  mounted  prism  was  difficult  to  remove  and  replace,  so  an  image  was  taken  with 
the  SF11  prism  mounted  on  the  camera,  but  the  camera  lifted  out  of  the  water.  Perforations  in  a  backlit  mask  are  stretched  into 
short  spectra  which  are  rendered  in  the  left  frame  as  small  apostrophes  in  this  monochrome  camera.  This  simulates  the  inverted 
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effect  of  the  water  with  no  SF11  prism.  Red  is  highest  and  strongest  because  of  the  low  color  temperature  of  the  illumination 
lamp.  After  the  camera  is  immersed,  a  new  image  is  then  gathered,  which  is  stretched  to  show  about  same  shape  (more  on  this  in 
the  section  on  distortion  below).  The  air  angle  of  the  target  is  about  15  degrees  from  the  horizon.  The  image  is  considerable 
improved,  although  it  is  still  a  trifle  overcorrected  because  of  the  finite  horizon  angle  and  the  slightly-too-large  specification  of 
the  14-degree  prism  angle. 
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Figure  4.  The  wedge-spectrum  correction  with  the  extra  SF1 1  prism  in  place.  Box  is  about  2  pixels  across. 


Figure  5.  Left:  Test  of  prism  in  air.  Right:  Test  of  prism  looking  from  water  into  the  air. 

B.  Distortion  Near  the  Horizon 

Much  of  the  literature  already  mentioned  has  discussed  the  severe  vertical  distortion  that  occurs  along  the  Snell  cone.  Figure  6 
is  a  depiction  of  a  wide  field  view  taken  along  the  Snell  cone,  with  circles  and  lines  superimposed  upon  it  to  guide  the  eye.  The 
Snell  cone  is  along  the  horizon  line.  Two  perspective  lines  are  drawn  radially  from  the  zenith.  Above  the  horizon  line  appears 
transmitted  light  from  above  the  water.  Below  the  horizon  line  is  reflected  light  and  the  base  of  the  aquarium.  In  the  inset  is  a 
detail  (taken  from  another  picture,  but  a  similar  one)  of  the  near-horizon  view.  This  time  the  horizon  line  has  not  been  drawn.  The 
squares  occupy  about  3.3  degrees  at  this  distance,  and  the  second  line  up  is  marked  as  6.6  degrees.  Below  the  3.3  degree  line  the 
horizon  is  inferred  but  not  seen.  This  image  was  gathered  when  there  was  no  chromatic  correction,  so  the  horizon  is  blurred. 
Clearly,  the  distortion  that  occurs  at  even  relatively  high  angles  is  severe.  From  taking  the  derivative  of  Snell’s  formula,  one  can 
derive  an  expression  for  the  vertical  magnification: 
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M(6)- 


^/l-cos2(#) 


2(*) 


(2) 


where  n  refers  to  the  respective  index  of  refraction  in  the  respective  medium,  #is  the  horizon  angle,  and  M  is  the  unnormalized 
magnification  (to  normalize  it  to  the  zenith  value,  just  multiply  it  by  (nwate/nair).  This  normalized  magnification  is  plotted  in  Fig.  7, 
and  it  shows  the  severe  compression  that  occurs  as  images  approach  the  horizon.  At  a  little  less  than  2  degrees,  for  example,  the 
flattening  is  20  to  1. 


Figure  6.  The  calm- water  aquarium  image  of  the  Snell  cone  (denoted  by  "horizon  line"). 
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Figure  7.  The  vertical  magnification  decreases  as  the  air  horizon  is  approached  until  the  image  is  compressed  into  a  horizontal  line. 
At  80  degrees  from  the  zenith  (10  degrees  from  the  horizon),  the  normalized  vertical  magnification  is 
only  about  a  fourth  of  the  horizontal  magnification. 
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Thus,  for  most  useful  imaging,  it  is  necessary  to  take  advantage  of  the  natural  disturbance  of  the  water’s  surface  to  occasionally 
tilt  up  the  section  of  surface  that  the  optical  system  views  through  so  that  locally  the  horizon  appears  elevated.  If  the  water  surface 
tilts  toward  the  direction  of  view  even  10  degrees  (and  the  next  wave  does  not  block  the  field-of-view),  the  flattening  lessens  to  4- 
to-1  and  the  image  may  be  discerned.  Using  resampling,  the  image  is  stretched  to  a  rough  semblance  of  the  air  view,  but  there  is 
an  associated  resolution  problem.  Resampling  cannot  create  information,  just  interpolate  it.  Such  images  have  a  resampling- 
induced  blurring  in  the  vertical  axis.  They  can  be  quite  sharp  horizontally,  but  the  stretching  creates  a  vertical  blur.  In  Fig.  8,  even 
such  narrow  horizontal  detail  as  the  gap  between  the  truck’s  front  fender  and  the  door  are  visible.  Vertical  details,  however,  such 
as  the  top  of  the  truck  bed,  show  only  the  jaggedness  from  excessive  resampling. 


Figure  8.  A  truck  passes  a  calm  pond  at  a  horizon  angle  of  10  degrees.  Picture  taken  from  a  few  cm  below  the  surface. 


The  solution  is  to  not  sample  an  image  unless  it  is  less  compressed.  This  can  be  done  by  waiting  until  the  surface  is  tilted  even 
more,  of  course,  but  that  is  not  a  very  good  solution.  The  more  tilted  a  surface  becomes  the  greater  amount  of  other  deformations 
that  are  less  advantageous.  Another  possibility  is  to  increase  the  focal  length  of  the  lens,  but  using  too  high  a  power  results  in  the 
field-of-view  being  outside  the  limits  of  the  frame  much  of  the  time.  It  also  increases  the  number  of  cameras  needed  to  view  in 
every  direction,  if  the  azimuthal  angle  of  the  target  is  not  known. 

Another  design  might  be  to  form  a  near  plane-parallel  window  by  tilting  the  window  of  the  camera  by  the  same  amount,  say  10 
degrees  (not  shown).  Then,  whenever  the  sea  surface  is  tilted  by  10  degrees,  the  window  of  the  camera  makes  a  plane-parallel  exit 
window.  This  method,  however,  when  extended  to  anything  like  a  significant  field-of-view,  takes  enormous  windows  to  allow  a 
camera  to  view  the  base  of  the  window  in  a  glancing,  sidelong,  manner.  It  is  not  a  workable  solution. 

The  only  reasonably  compact  design  is  to  make  an  anamorphotic  lens,  also  called  an  “anamorphic”  lens.  Now  these  can  be 
constructed  either  on  the  Chretien  pattern[9],  basically  using  cylindrical  or  toroidal  lenses  to  make  what  amounts  to  a  Galilean 
telescope  in  one  axis,  or  on  the  prism  anamorphoser  principle,  often  used  in  stretching  or  compressing  laser  beams  in  one  axis. 
The  prism  combinations  tend  to  be  bulky  and  thick,  and  unlike  its  use  in  lasers,  this  prism  anamorphoser  must  be  achromatic. 

An  impediment  may  be  converted  to  an  advantage  in  our  case  by  noting  that  we  already  have  the  first  element  of  the 
anamorphic  prism-pair  in  the  seawater  wedge.  Thus  we  need  only  add  one  more  prism  to  achieve  stretching.  Such  a  design  is 
shown  in  the  Fig.  9  ZEMAX  layout.  It  is  a  two-prism  design,  but  only  one  prism  need  be  built  and  carried  within  the  camera, 
with  the  forward  prism  being  the  wedge  of  seawater  outside  the  camera  and  the  second  prism  being  carried  in  front  of  the  iris. 

The  pair  of  prisms,  one  virtual  and  outside  the  device,  and  the  other  inside  the  device,  are  collectively  achromatic  in  the  same 
sense  as  the  previous  design.  They  expand  the  vertical  axis  only  by  a  factor  of  about  1.35  in  this  experimental  instrument,  but  the 
ratio  could  be  adjusted  by  changing  the  angles  and  materials  of  Fig.  9.  The  primary  limit  to  increasing  the  magnification  of  prism 
anamorphosers  is  that  the  final  exit  angle  on  the  last  prism  limits  the  vertical  field-of-view,  and  the  greater  the  stretch,  the  more 
limited  that  field-of-view  is. 

This  anamorphoser  was  built  for  an  experimental  laboratory  camera  and  tested.  The  two  pictures  of  Fig.  10  were  taken  in  the 
afternoon  and  morning,  which  accounts  for  the  different  shadows.  The  upper  one  has  the  anamorphic  prism  fitted,  and  each  was 
taken  during  calm  conditions.  They  depict  a  door  and  shuttered  window  at  the  end  of  a  floating  pier.  The  entrance  pupil  of  the 
camera  is  a  few  inches  below  the  surface  of  the  water,  and  these  images  were  only  mildly  compressed  at  a  horizon  angle  of  about 
20  degrees.  The  door  spring-dashpot  and  the  railing  are  pointed  out,  and  at  least  in  these  two  images,  the  anamorphic  system  leads 
to  image  improvement. 
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Figure.  9.  Experimental  anamorphic  design.  The  small  prism  is  Schott  N-F2 


Figure  10.  Results  of  the  experimental  anamorphotic  configuration. 


C.  Blurring  from  Ref  r action 

Clearly,  the  optical  difficulties  in  sections  A  and  B  are  important,  but  they  are  straightforwardly  solved.  The  difficulty  that 
creates  the  greatest  problem  is  optical  refraction  at  the  interface  between  the  two  media.  The  surface  of  the  water  is  distorted  by  a 
spectrum  of  waves  extending  from  long  gravity  waves  to  short  capillary  waves.  This  structural  distortion  is  translated  into  optical 
aberrations  on  the  bundle  of  light  entering  the  entrance  pupil  of  the  lens.  If  the  pupil  is  extended  up  along  the  parallel  ray  bundle 
it  intercepts  the  strongly  refracting  surface  in  a  small  oval  shape.  If  the  pupil  is  large,  there  is  much  structure  on  this  surface.  If  the 
pupil  is  small,  there  is  less  structure.  Thus,  small  pupils  have  a  greater  probability  of  being  simple  windows. 

By  diminishing  the  size  of  the  pupil  further,  a  condition  will  be  reached  where  the  only  remaining  structure  is  the  tilt  given  the 
wave  front  by  the  refraction.  Yet  still  the  scene  is  not  visible  for  a  highly  structured  surface.  Only  for  relatively  smooth  surfaces  is 
an  image  discerned.  (The  exposure  is  assumed  fast  enough  to  render  motion  negligible.) 

The  cause  is  anisoplanatism,  or  formally  the  lack  of  space  invariance  in  the  impulse  response  of  an  optical  system  [10]. 
Anisoplanatism  is  simply  the  lack  of  correlation  of  the  pupil  aberration  from  one  look  angle  to  another.  As  the  pupil  is  drawn  up 
to  the  refracting  surface,  a  single  wave  front  shape  on  that  pupil  influences  the  whole  field-of-view  but  any  significant  separation 
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results  in  the  situation  of  a  (these  situations  are  sketched  for  a  vertical  field-of-view  in  Fig.  11).  Thus,  optimum  conditions  prevail 
when  the  pupil  is  close  the  surface,  although  this  is  a  seldom-realized  condition  for  any  outdoor,  real-world  water  surface.  Indeed, 
it  is  only  for  calm  water  that  isoplanatism  can  be  precisely  achieved  at  all.  Most  commercial  camera  lenses,  which  contain  the 
virtual  position  of  the  entrance  pupil  far  behind  the  front  surface  of  the  lens,  also  preclude  this  form  of  isoplanatism.  Keeping  the 
pupil  as  near  as  possible  to  the  surface  and  keeping  the  prisms  small  were  the  reasons  behind  using  the  stop-forward  lens  designs 
presented  earlier  in  this  paper. 


Figure  11.  a)  Anisoplanatism  caused  by  different  refracting  regions  on  the  surface,  b )  Isoplanatism  is  recovered  when  the  pupil  is  at  the  surface. 


Anisoplanatism  has  been  discussed  in  depth  for  atmospheric  propagation  by  Fried  [11],  chiefly  having  to  do  with  astronomical 
adaptive  optics.  Fried  and  others  have  also  discussed  “lucky”  imaging,  or  constructing  a  composite  image  from  a  carefully 
selected  subset  of  images  [12] [13],  In  Snell-cone  optics,  lucky  imaging  takes  on  a  new  form.  It  is  not  merely  waiting  for  the  view 
to  clear,  but  waiting  for  the  view  to  be  possible.  Fig.  12  shows  the  problem  with  viewing  near  the  horizon.  The  obscured  or  total- 
internally-reflected  cases  are  a  significant  fraction  of  the  possibilities,  depending  on  the  viewing  angle  and  the  deformation  of  the 
surface,  of  course.  In  this  experiment,  lucky  image  sorting  was  done. 


Figure  12.  Horizon  views  must  be  sorted  from  a  sequence  of  images. 


A  patch  of  the  image  that  is  isoplanatic  is  not  the  only  condition  that  is  required  for  successful  imaging.  The  wave  front 
emerging  from  the  entrance  pupil  must  be  fairly  flat  before  good  imaging  is  possible.  If  it  is  not  (see  Fig.  11  b),  the  image  may  be 
approximately  isoplanatic  but  still  not  sharp.  In  the  lucky  exposures  method,  if  the  wave  front  is  accidentally  flat  or  very  slightly 
curved  over  at  least  a  part  of  the  image,  it  necessarily  has  a  large  isoplanatism  patch.  If  it  not  occluded  as  some  of  the  examples 
are  in  Fig.  12,  we  will  see  it. 

We  can  expand  the  wave  front  on  the  generalized  pupil  by  a  set  of  complete  orthogonal  polynomials,  where  one  form  of  the 
low  orders  is  presented  in  Table  1  [14];  p  is  the  normalized  radius  of  the  pupil  and  cp  is  an  arbitrary  azimuth  angle  in  the  plane  of 
the  pupil.  These  terms  are  associated  with  a  set  of  coefficients,  and  if  taken  to  infinite  order  in  p,  completely  describe  the  wave 
front.  In  practice,  the  coefficients  diminish  for  higher  powers  of  p. 

This  orthogonal  set  was  first  defined  on  the  unit  circle  by  Frits  Zernike,  who  called  them  circle  polynomials.  Note  that  the 
lowest  orders,  namely  piston  and  distortion,  are  still  flat  and  hence  will  not  blur  the  image  (piston  is  mostly  a  computational 
device).  The  lowest-order  blurring  aberrations  are  defocus  and  the  two  axes  of  astigmatism.  If  the  pupil  is  kept  small  enough,  we 
can  approximate  this  expansion  by  its  first  six  terms,  truncating  the  series  at  Z6. 

TABLE  I 

AN  ORTHOGONAL  DESCRIPTION  OF  ABERRATIONS  INDUCED  ON  AN  ENTRANCE  PUPIL. 


Term 

Zernike  polynomial 

Name 

Zi 

1 

piston 

z2 

2p  cos(cp) 

x-distortion 

Z3 

2p  sin((p) 

y-distortion 

Z4 

V3 (2p2-l) 

defocus 

Z5 

V6  p2  sin(2(p) 

45-135  deg  astigmatism 

Z6 

V6  p2  cos(2(p) 

0-90  deg  astigmatism 

Z7 

V8  (3p2— 2)  p  sin((p) 

y-coma  (drumhead) 

Z8 

V8  (3p2— 2)  p  cos((p) 

x-coma  (drumhead) 

Z9 

V8  p3  sin(3(p) 

30-150  deg  trefoil 

Zio 

V8  p3  cos(3(p) 

0-120  deg  trefoil 

Zn 

V5  (6p4  —  6p2+  1) 

spherical  aberration 
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Note  that  both  defocus  and  astigmatism  terms  are  proportional  to  p2,  meaning  that  in  one  or  two  dimensions,  they  follow  the 
first  term  in  the  expansion  of  a  spherical  deformation.  Given  the  lens  focal  length  (20  mm)  and  the  aperture  dimension  (2.8  mm), 
the  blurring  or  stretching  at  the  focal  plane  may  be  readily  calculated  as  the  values  in  Table  II.  As  long  as  the  radius  of  the  sea 
surface  is  beyond  0.5  m,  then  the  destructive  effects  of  sea  curvature  is  relatively  small,  but  it  quickly  becomes  severe  at  small 
radius.  It  was  witnessed  during  the  test  that  when  a  capillary  wave-train  went  over  the  camera,  little  was  visible.  The  probable 
reason  was  that  the  radius  of  curvature  was  too  small,  and  hence  the  induced  blurring  was  too  great. 

One  may  also  use  the  sum  of  the  three  circle  polynomials  Z4  through  Z6  as  a  kernel  for  a  deconvolution  algorithm,  searching 
over  a  three-dimensional  coefficient  space  for  the  best  fit  to  a  sharpness  metric  such  as  an  edge  finder  or  entropy  algorithm. 

TABLE  II 

BLURRING  IN  MICROMETERS  DUE  TO  A  SEAWATER  SURFACE  OF  RADIUS  R 


R(mm) 

blur(jlim) 

pixels 

100 

191 

18.0 

200 

95 

9.0 

300 

64 

6.0 

400 

48 

4.5 

500 

38 

3.6 

600 

32 

3.0 

700 

27 

2.6 

800 

24 

2.2 

900 

21 

2.0 

1000 

19 

1.8 

III.  ACTUAL  IMAGES 


A.  The  Camera 

The  camera  is  a  PhotonFocus  Hurricane™  high  dynamic-range  linear-log  CMOS  type.  It  is  important  for  this  application  that 
the  camera  be  a  global- shutter  type  and  that  it  not  have  any  progressive  scan  or  interline  artifacts.  It  is  capable  of  extremely  fast 
exposures  (less  than  0.1  ms)  and  because  the  pixels  are  10.6  by  10.6  pm,  has  reasonable  sensitivity.  It  renders  the  image  in  4096 
levels. 

B.  The  Test  Bed 

The  test  bed  (Fig.  13)  is  a  3-channel  unit  that  uses  the  achromatic  wedge  but  not  the  anamorphoser.  The  anamorphic  attachment 
was  viewed  as  too  risky  at  the  time  this  unit  was  designed.  It  is  connected  to  shore  computers  by  a  fiber-optic  cable  and  a  power¬ 
carrying  copper  wire.  It  was  not  buoyed  naturally,  but  attached  between  pilings. 


Figure  13.  The  Arete  3 -channel  test-bed  unit. 


C.  The  Site 

The  site  was  an  abandoned  campground  on  Choctawhatchee  Bay  at  Hammock  Point.  Hurricanes  have  ruined  its  pier  and  there 
are  pilings  standing  about  with  separations  of  1.5  to  3  m.  There  is  about  6  km  of  open  bay  in  front  of  the  test  side.  Wind  was 
between  6  and  12  kn.  There  was  no  surf  action. 


D.  The  Targets 

The  targets  were  various  vehicles,  resolution  targets,  and  targets  of  opportunity  ranging  from  0  degrees  to  about  5  degrees  off 
the  horizon.  They  are  shown  in  later  pictures. 
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IV.  DISCUSSION 


The  small  amount  of  data  that  we  have  been  able  to  analyze  thus  far  have  taught  us  that  this  is  a  problem  of  sorting  through 
massive  amounts  of  information  for  those  tiny  snippets  of  image  which  contain  enough  detail  to  be  profitably  processed. 

The  lucky  imaging  approach  takes  advantage  of  the  fact  that  the  undulating  water  surface  occasionally  provides  a  surprisingly 
sharp  view  of  the  scene — perhaps  once  in  hundreds  of  frames.  Finding  the  lucky  frame  can  be  a  time  consuming  process,  so 
software  is  needed  to  automatically  select  the  useful  frames  from  very  large  data  sets.  An  image  processing  graphical  user 
interface  was  created  that  allows  easy  visualization  of  the  data  and  contains  a  sorting  algorithm  to  score  the  image  quality.  Fig.  14 
is  a  sequence  of  images  that  occurred  over  only  0.2  s.  The  scene  oscillated  between  bad  and  good  in  times  less  than  the  frame  rate 
of  20  Hz. 


Figure  14.  Four  consecutive  frames.  Only  the  third  is  reasonably  interesting. 


In  the  course  of  analyzing  data  collected  under  a  variety  of  conditions,  it  has  become  obvious  that  it  is  very  difficult  to  design  a 
sorting  metric  that  will  be  effective  in  every  circumstance.  The  first  sorting  metric  was  developed  using  data  collected  in  a  pond 
with  an  earlier  version  of  the  sensor.  The  water  was  very  calm  so  the  images  were  fairly  stable  and  underwent  relatively  modest 
fluctuations  in  sharpness.  In  some  sense,  all  pond  images  are  lucky  frames  since  they  all  contained  more  or  less  recognizable 
objects.  The  initial  step  of  the  developed  metric  is  processing  the  images  with  a  Sobel  filter  and  calculating  the  cumulative 
distribution  function  (CDF)  of  the  filtered  images.  The  image  score  was  calculated  as  the  value  of  the  CDF  at  a  fixed  level  of  the 
filtered  images.  This  metric  succeeded  in  finding  the  sharpest  images  even  under  quite  varied  scenery,  but  depended  on  having  a 
relatively  stable  scene  with  which  to  work,  i.e.  calm  water,  and  also  depended  on  high  contrast  targets. 

A.  The  Bay  Sort  Metric 

Sorting  the  Chocktawhatchee  Bay  data  is  considerably  more  challenging  due  to  the  increased  wave  content  in  the  imagery.  This 
increase  lead  to  a  development  of  a  new  sorting  algorithm.  The  improved  sorting  algorithm  is  initiated  by  finding  subregions  of 
the  image  that  are  likely  to  contain  sharp  boundaries  between  light  and  dark  patches  of  water.  This  is  accomplished  by  calculating 
brightness  histograms  of  the  image  subregions,  like  those  shown  in  Fig.  15,  and  looking  for  indications  of  a  multimodal 
distribution  and  higher  than  average  variance.  Subregions  are  flagged  as  “interesting”  based  upon  a  complex  formula  involving 
the  properties  of  the  histograms’  variance  and  the  number  of  distinct  peaks.  The  final  sort  metric  is  calculated  based  upon  the 
difference  between  the  number  of  adjacent  “interesting”  subregions  forming  the  longest  row  and  the  number  of  distinct  connected 
“interesting”  subregions.  This  sort  metric  tends  to  favor  images  with  a  distinct,  elongated  horizon  and  reject  images  with 
“interesting”  subregions  randomly  scattered  across  the  image,  namely  the  basket  pattern  of  capillary  waves.  (See  Fig.  16.) 


0  5  10  15  20  0  5  10  15  20 


Figure  15.  Two  histograms:  (left)  “uninteresting,”  and  (right)  “interesting.” 
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Figure  16.  (Left)  Uninteresting  frame  vs.  (Right)  Interesting  frame. 


The  effectiveness  of  the  sorting  metric  was  assessed  by  counting  the  number  of  lucky  frames  in  a  variety  of  datasets  and 
determining  the  number  of  lucky  frames  found  near  the  top  of  the  sorted  list.  A  subjective  judgment  was  necessary  to  determine 
which  frames  were  considered  lucky,  and  owing  to  the  tedium  of  hand- selecting  lucky  frames,  only  a  few  subsets  of  the  image 
sequences  were  analyzed. 

Table  III  summarizes  the  results  of  this  analysis.  Two  datasets  from  the  bay  test  were  analyzed,  one  with  5-inch  waves  and  the 
other  with  6-inch  waves  (peak-to-peak  amplitude).  The  number  of  lucky  frames  was  tabulated  in  three  subsets  of  the  imagery:  the 
first  20%  of  the  frames,  a  set  of  1000  contiguous  frames  drawn  from  the  3rd  quartile,  and  the  last  1000  frames.  The  images  were 
ordered  both  chronologically  and  according  to  the  ranking  assigned  by  our  sort  metric. 

Analyzing  the  8400  chronologically  ordered  frames  in  the  5-inch  wave  dataset  yielded  a  total  of  57  lucky  frames.  Assuming  the 
lucky  frames  are  more-or-less  evenly  distributed  chronologically,  the  5-inch  data  set  therefore  contains  approximately  220  lucky 
frames  out  of  32000,  about  a  0.7%  lucky  frame  rate.  When  the  sorting  metric  is  applied,  a  total  of  229  lucky  frames  are  found  in 
the  top  20%,  a  number  consistent  with  the  estimated  total  number  of  lucky  frames,  and  none  are  found  in  the  other  sorted  data 
subsets.  This  verifies  that  the  sort  metric  successfully  moved  the  bulk  of  the  lucky  frames  to  the  front. 

In  the  6-inch  dataset,  a  total  of  13  lucky  frames  were  found  in  a  total  of  5400  chronologically  ordered  frames.  This  leads  to  an 
estimate  that  about  40  lucky  frames  are  present  in  the  entire  17000-frame  dataset,  about  a  0.2%  lucky-frame  rate.  In  the  sorted 
frames  a  total  of  20  lucky  frames  occurred  in  the  top  20%  and  none  in  the  other  two  subsets.  The  estimate  for  the  total  number  of 
lucky  frames  suggests  there  may  be  more  found  below  the  top  20%,  but  once  again  the  sort  metric  performed  well  in  selecting 
lucky  frames.  Based  upon  the  pond  data  and  the  two  bay  test  datasets,  a  trend  emerged  that  show  lucky  frames  become 
increasingly  rare  with  higher  amplitude  waves. 

The  sort  metric  designed  using  Chocktawhatchee  Bay  test  data  was  applied  to  the  pool  test  data  taken  during  the  camera 
checkout,  which  is  considerably  different  because  the  pool  has  many  more  high  elevation  objects  (trees,  house,  targets,  etc.),  and 
did  not  perform  well.  This  shows  that  it  is  extremely  challenging  if  not  impossible  to  design  a  single  sort  algorithm  that  will  select 
lucky  frames  under  all  circumstances,  but  the  metric  designed  for  the  Bay  test  data  should  work  well  whenever  the  objects  in  the 
scene  are  confined  to  low  elevations.  A  secondary  sort  metric  can  be  developed  and  applied  to  the  higher  elevation  target 
allowing  one  tool  to  work  in  both  instances  depending  upon  the  task  at  hand. 

The  Chocktawhatchee  Bay  data  are  notable  for  their  lack  of  surf  action.  An  experiment  in  the  presence  of  surf  is  suggested, 
since  it  regularly  lifts  the  surface  toward  the  shore,  and  necessarily  has  a  long  wavelength. 


TABLE  III 


ANALYSIS  OF  SORTING  METRIC 

Total  number  of  frames 

32000 

17000 

Estimated  lucky  frame  fraction 

0.7% 

0.2% 

Sorted  Chronological 

Sorted 

Chronological 

Number  of  lucky  frames  in  top  20% 

229/6400  9/6400 

20  / 3400 

7 / 3400 

Number  of  lucky  frames  in 
contiguous  1000  frames  in  3rd 

0/1000  8/  1000 

0/1000 

1  / 1000 

quartile 

Number  of  lucky  frames  in  last  1000 

0/1000  40/1000 

0/1000 

5/1000 
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B.  Sub-regions 

There  is  reason  to  hope  that  sub-regions  of  the  image  can  be  collected  to  form  larger  images.  In  Fig.  17,  we  see  how  individual 
regions  can  show  clarity  even  if  the  rest  of  the  image  is  blurred  beyond  recognition.  In  a )  we  see  the  in-air  field-of-view,  except 
that  the  leftward  piling  is  a  little  far  to  the  left  and  no  crossing  rope  is  visible  to  the  water  camera.  Frame  b)  shows  that  if  one  is 
very  patient,  one  can  see  a  recognizable  scene  clear  across  the  frame.  This  is,  in  fact,  the  field  of  two  cameras  stitched  together  to 
show  the  building  at  the  right.  By  subdividing  the  frame  to  sub-regions,  we  can  perhaps  make  a  frame  out  of  pieces.  There  is  one 
troubling  disadvantage  with  this  procedure,  however,  and  that  is  the  seemingly  random  position  that  resolved  areas  end  up.  That  is 


Figure  17.  Image  frames  occasionally  show  glimpses  through  the  blur  regions,  but  only  in  small  gaps. 
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indicated  by  the  connections  indicated  to  obviously  the  same  items  in  the  separate  fields-of-view.  The  left-right  position  of  these 
resolved  areas  of  the  target  are  only  approximately  the  same  from  frame  to  frame.  This  makes  sense  if  we  consider  that  the  values 
of  the  non-blurring  Zernike  tilt  coefficients  of  Z2  and  Z3  vary  quite  a  bit  even  for  these  fairly  uniform  surfaces. 

C.  More  Work  for  Different  Environments 

The  success  or  lack  of  it  of  the  Chocktawhatchee  Bay  test  perhaps  depended  on  the  direction  (it  came  from  the  rear  quarter 
compared  to  the  look  angle)  and  uniformity  of  the  wind,  and  also  the  presence  of  the  support  pilings  as  generators  of  air 
turbulence.  In  an  early  St.  Andrews  Bay  test  of  a  laboratory  camera,  other  performance  values  were  witnessed.  The  laboratory 
camera  was  held  relatively  far  from  its  supports,  and  hence  did  not  have  a  nearby  source  of  capillary  waves,  in  the  snippet  of 
imagery  represented  by  the  frames  in  Fig.  18,  the  rate  of  interesting  frames  was  at  least  1  in  50.  Clearly  more  work  is  needed  in 
different  environments  before  deciding  that  lucky  imaging  produces  too  low  a  frame  rate. 


Figure  18.  Frames  gathered  under  different  conditions.  The  bottom  of  the  lower  frame  is  less  than  1  degree  high. 


V.  CONCLUSIONS 

1)  The  likeliest  method  of  near-horizon  viewing  is  to  use  a  small  entrance  pupil  and  to  place  it  as  near  to  the  surface  as  possible. 

2)  Correction  of  wedge  prism  color  is  necessary  in  near-horizon  air-to- water  viewing.  Owing  to  the  crown-like  nature  of  seawater 
dispersion,  this  correction  is  easy,  requiring  no  more  than  a  thin  dispersive  prism. 

3)  The  distortion  that  occurs  when  light  is  coming  from  the  horizon  may  be  mitigated  by  waiting  for  a  surface  deformation  that 
leans  toward  the  viewed  horizon.  It  may  be  further  lessened  by  using  anamorphotic  lenses. 

4)  The  method  of  lucky  imaging  may  be  used  to  construct  a  horizon  image.  This  process  is  extremely  inefficient,  however,  which 
leads  to  suggested  further  work  on  methods  to  speed  it  up,  as  well  as  investigations  in  other  environments. 
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